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Abstract 

We consider a generalization of the Hopfield model, where the entries of pat- 
terns are Gaussian and diluted. We focus on the high-storage regime and we in- 
vestigate analytically the topological properties of the emergent network, as well 
as the thermodynamic properties of the model. We find that, by properly tuning 
the dilution in the pattern entries, the network can recover different topological 
regimes characterized by peculiar scalings of the average coordination number 
with respect to the system size. The structure is also shown to exhibit a large 
degree of cliquishness, even when very sparse. Moreover, we obtain explicitly the 
replica symmetric free-energy and the self-consistency equations for the overlaps 
(order parameters of the theory), which turn out to be classical weighted sums 
of "sub-overlaps" defined on all possible sub-graphs. Finally, a study of critical- 
ity is performed through a small-overlap expansion of the self-consistencies and 
through a whole fluctuation theory developed for their rescaled correlations: both 
approaches show that the net effect of dilution in pattern entries is to rescale the 
critical noise level at which ergodicity breaks down. 



1 Introduction 



In the last two decades, the application of statistical mechanics for the investigation of 
complex systems has attracted a growing interest, also due the number of applications 
ranging from economic (lj and social sciences [2], to biology and neurobiology |3J, to 
theoretical immunology [I] and even to computer science [5] and machine learning [6]. 
As a consequence, the introduction and the design of always deeper models along with 
the development of suitable techniques for their analysis becomes more and more impor- 
tant for theoretical physicists and mathematicians involved in the field. In particular, 
attention has been devoted to the interplay of complex topology and critical behaviour, 
evidencing the strong relation between the network structures and the thermodynamics 
of statistical mechanics models [7] . 

In this paper we make a step forward in this direction, by analysing a Hopfield model 
[5J where pattern entries can be either extracted from a Gaussian distribution or 
set equal to zero. More precisely, entries are drawn from a normal distribution A/"[0, 1] 
with a probability (1 + a)/2 or set equal to zero with probability (1 — a)/2, where 
a G [—1, +1] is a tunable parameter controlling the degree of dilution of patterns. We 
focus on the high-storage limit, namely the amount of patterns L is linearly diverging 
with the system size N, i.e. L = a>N. 

This kind of " analogue" neural networks has been intensively studied on fully con- 
nected topologies (see for instance [TQ1 HU [TJ1 H3]) and further interest in the model 
lies in its peculiar "soft retrieval" as explained for instance in [T3j. 

Here, we first study the topological properties of the emergent weighted network, 
then we pass to the thermodynamic properties of the model. 

In particular, we calculate analytically the average probability for two arbitrary 
nodes to be connected and we show that, by properly tuning a, the network spans 
several topological regimes, from fully connected down to the percolation threshold. 
Moreover, even if the network is very sparse, it turns out to display a large degree of 
cliquishness due to the Hebbian rule underlying its couplings. The coupling distribution 
is also explicitly calculated and shown to be central and with extensive variance, as 
expected. 

From a thermodynamic perspective, using an exact Gaussian mapping, we prove 

that this model is equivalent to a bipartite diluted spin-glass (or to a Restricted Boltz- 

man Machine [15] retaining a cognitive system perspective [IS]), whose parties are made 

up by binary Ising spins and by Gaussian spins, respectively, while interactions among 
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them, if present, are drawn from a standard Gaussian distribution Af[0, 1]; of course, 
there are no links within each party. The size of the two parties are respectively N, for 
the Ising spins, and L, for the Gaussian ones, and the dilution in the Hopfield pattern 
entries corresponds to standard link removal in this bipartite counterpart. 
Then, extending the technique of multiple stochastic stability (developed for fully con- 
nected Hopfield model in [13] and for ferromagnetic systems on small-world graphs in 
[T7] ) to this case, we solve its thermodynamics at the replica symmetric level. Once 
introduced suitably order parameters for this theory, we obtain an explicit expression 
for the free-energy density that we extremize with respect to them to obtain the self- 
consistencies that constraint the phase space of the model. As in other works on diluted 
networks [TTl IT8| [T9] , the order parameters are two (one for each party) series of overlaps 
defined on all the possible subgraphs through which the network can be decomposed. 
A study of their rescaled and centered fluctuations allows to obtain the critical surface 
delimiting the ergodic phase from the spin-glass one. The same result is also recovered 
through small overlap expansion from the self-consistencies; the agreement confirms the 
existence of a second order phase transition (3], |20] . 

The paper is organized as follows: in Section 2 the model is defined with all its related 
parameters and variables, while in Section 3 its topological properties are discussed. 
Section 4 deals with the statistical mechanics analysis while in Section 5 fluctuation 
theory is developed. Section 6 is left for a discussion and outlooks. 



2 The model 

Given N Ising spins <7j = ±1, i G (1, N), we aim to study a mean-field model whose 
Hamiltonian has the form 

1 N 

H = -— Y^JijWj, (1) 

ij 

where the couplings are built in a Hebbian fashion [21 J [Sj as 

JH = Y,#&> ( 2 ) 
11=1 

and D is a denominator whose specific form is discussed in Section 3. In fact, in general, 

as the coordination number may vary sensibly according to the definition of patterns 

£, in order to ensure a proper linear scaling of the Hamiltonian ([I]) with the volume, D 
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has to be a function of the system size N and of the parameters through which patterns 
£ are defined. 

We consider the high-storage regime [22], such that, in the thermodynamic limit 
(i.e. iV — y oo), the following scaling for the amount of stored memories (patterns) is 
assumed 

lim 4 = " e (3) 

even though we use the symbol a for the ratio between the number of patterns and 
the system size also at finite N, bearing in mind that the thermodynamic limit has to 
be performed eventually. 

The quenched entries of the memories £f are Gaussian and diluted, namely they are 
set to zero with probability (1 — a)/2, while, with probability (1 + a)/2, they are drawn 
from a standard Gaussian distribution: 

p{$) = (^) m) + ( 1 ^)^ [ o,i ] (£f) • (4) 

The parameter a can in principle be varied in the range a £ [—1, 1], and, in general, 



small values correspond to highly diluted regimes. As proved in Section a scaling 
law for this parameter has to be introduced in order to avoid the topology to become 
trivial in the thermodynamic limit. Thus, we consider the following scaling 

a = - 1 + Jfe > ( 5 ) 

where 9 determines the topological regime of the network, while 7 plays the role of 
a fine tuning within it. More precisely, 7 £ (0, 2] and, of course, for 7 = we get 
P = 5(£f), that is, there is no network, so we discuss only the case 7 > 0. Finally, 
notice that fixing 8 = and 7 = 2 yields to a = 1, corresponding to the standard 
analogue Hopfield model 



3 Topological analysis 
3.1 Coupling distribution 

Let us consider the definition of the coupling strength in Equation ([2]): the probability p 
that the /i-th term is zero corresponds to the probability that at least one between 
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£f and £j* is zero, which is 



P 



1 — a\' / 1 — a\ / 1 + a 
~t~ 2 



3 - a 2 - 2a _ / I + o 



(6) 



while its complement is the probability that a Gaussian number is drawn for both 
entries, that is (1 — p) — [(1 + a)/2] 2 . Thus, the probability that the link connecting i 
and j has strength can be written as 



fc=i ^ ^ v 



(7) 



P' 



P" 



fc=i 

L 



fc=i 



^ 2^(l + /2)*/2 



where to simplify the notation we defined /(&) = — p) k (^) and -P& ( £f £J = 

Jij) is the probability that k pairs of Gaussian entries, pairwise multiplied, sum up to 
Jij, namely 

e ^ = j « - ) = / n ^ e ^ - j < 



dl 
2^ 



up 2 «r> 2 



i/=0 



y=0 

I 2 

e 2 e 2 
27 v 7 ^ 



2vr(l + / 



2^/2 ' 



(9) 



From Equation (|8j) one can easily specify the characteristic function of the coupling 
distribution 



F(l) 



/+oo n 
e ilJ P{J)dJ = p L + J2~ 
00 fc=i I 1 



/(*) 



1 



+ Z 2 ) 



k/2 



1 + 



i + zy r 
p + / 2 p 



(10) 



where we dropped the indices i and j, due to the arbitrariness of the couple of nodes 
considered. From F(l) it is possible to obtain all the momenta by simple differentiation. 
For instance, first and second moment read respectively as 



E[Jy] = (~i 



dF(l) 
. dl 



0. 



(11) 



1=0 



-i) 



L{l-p) = L 



1=0 



l + a 



cry 



N 



1-29 



(12) 



Now, for fixed a and a, we expect that J, being a sum of Gaussian variables, is also 
normally distributed (except the point J = 0), at least for large N. Indeed, numerical 
simulations confirm that the distribution P(J) converges in the thermodynamic limit 
(L — > oo) to a Gaussian distribution with zero mean and variance given by Equation 
(12) (see Figure [TJ, except for the point J = which will be discussed in the following 
section. 




Figure 1: Coupling distribution P(J) for L — 10 (left panel, blue), L = 20 (left panel, red), L = 30 
(left panel, green) and L = 1000 (right panel). Circles represent the coupling distribution calculated 
according to Eq. ^1 while the continuous lines represent normal distributions with momenta given by 



Eq. 11 and Eq. [12j as the thermodynamic limit is approached, the agreement gets better and better. 
Notice that, when L grows, the divergence in J = becomes weaker. Only the positive values of J are 
considered due to the symmetry. 



3.2 Link Probability and topology regimes 

Let us consider the bare topology. The quantity of interest is the average link probability 
-Piink: 

Pi lnk = l-P(J = 0). (13) 

Looking at Equation (m), in principle P(J = 0) has two contributions: one from the 

delta function and one from the sum over k random numbers, but the latter has a null 
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measure in the limit L — > oo. To show this we consider the second term in Equation 
([7| and we calculate its measure over the interval Jy G [— e, +e], highlighting for clarity 
the term k = 1: 



J ~ e fc=l V 

= p 1 - 1 (1 - p)L f P 1 {r)dr + (l) P L ~ k C 1 - ?)* f P *( r ) rfr • ( 14 ) 

./-e fe=2 V / J-e 

In fact, we notice that P\(r) has a weak divergence in r = and its integral scale as 
~ elog(e), so that the divergence is suppressed by the prefactor in the limit L — > oo, so 
that the first term in Eq. [14] is vanishing. As for Pk>i(r), its integral is non-diverging 
and can be upper bounded [|] to show that the second term is also negligible in the limit 
L — > oo. 

Hence, in the thermodynamic limit, Jjj = only if £f£j* = 0, for any /x, namely 

3 -a 2 -2a x L 



P(J IJ = 0)=P L =[ - A ) ■ (15) 



Now, looking at (13) and (15) in the thermodynamic limit, it is clear that, if we 
consider a as finite and constant, only two trivial topologies can be realized. In fact, 
if a = —1, Punk is zero and the system is fully disconnected, while, with a > — 1, 
Piink tends to one exponentially fast with the system size, and the graph becomes fully 
connected. 



Nevertheless, with the scaling ([5]), 

2 \ aN 



link 



7 



AN 29 



e SP^, (16) 



where the last expression holds for large N and 7 G (0,2]. Now, by tuning the value 
of 9, we realize different topological regimes; within each regime the parameter 7 acts 
as a fine tuning. Following a mean-field approach, namely just focusing on the average 



1 From the the two inequalities fl e Pk(r)dr < 2ePk(0), Pk{0) < c 1 "^- 1 (with c a constant), it 
follows 

k=2 V ' J - e k=2 V J 

which goes to zero in the limit L — > 00. 



link probability, we can distinguish: 

/ 2\ aN 

• 9 = : P link = 1—11 — ^-1 —7-1. Fully Connected graph, with average degree 
equal to the system size {z = N — 1). 

The coupling distribution converges to the Gaussian one with variance Var[J] oc 
N, as in the with Sherrington-Kirkpatrick model. 

2 

• < 9 < 1/2: Pii nk = 1 - e ~^ Nk -» 1 (where < k < 1). Fully Connected graph, 
with average degree equal to the system size (z = N — 1). 

The coupling distribution converges to the Gaussian one with Var[J] oc N k . 



• 9 = 1/2: Pii n k — = const. The link probability is finite and the average coordi- 
nation number is linearly diverging with the system size, namely z = a^ 2 {N — 1) = 
0(N), and Var[J] cx cost. 

• 1/2 < 9 < 1: Pi ink = 1 - e -^ Nk ~ (where -1 < k < 0). Extreme Di- 
luted Graph, characterized by a sublinearly diverging average coordination num- 
ber, z = OiN 1 -*), and Var[J] cx N k . 

at 2 2 

• 9 = 1 : Punk = 1 — e~ 4at ~ — y o. Finite Coordination Regime with z = a7 2 /4, 
and Var[J] cx 1/JV. 

• 9 > 1 : Punk = 1 - e"^* ~ -)■ (where k < -1). Fully Disconnected 
Regime with coordination number vanishing for any choice of a and 7. The 
variance of the coupling distribution is vanishing superlinearly with N. 

A contour plot of P un k as a function of 7 and 9 is shown in Figure [2j 
3.3 Small-world properties 

Small-world networks are characterized by two main properties: a small diameter and a 
large clustering coefficient, namely, the average shortest path length scales logarithmical 
(or even slower) with the system size and they contain more cliques than what expected 
by random chance [23J . The small- world property has been observed in a variety of real 
networks, including biological and technological ones [24]. 

First, we checked that, in the overpercolated regime, the structures considered here 
display a diameter growing logarithmically with N, as typical for random networks [7] . 
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log(0) 



Figure 2: The main figure represents the contour plot of Pn n k (see Equation 16 ) as a function of 7 and 
of log 9. The dashed, vertical line corresponds to 9 = 1/2 and demarcates the onset of a disconnected 
regime. The inset represents the degree distribution P(z) as a function of the normalized number of 
nearest neighbors; three values of 9 are considered as specified. Notice that, as expected, larger values 
of 9 yields to sparser graphs. Both figures refer to systems made up of N = 6000 nodes, with a = 0.05 
and 7 = 1. 
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As for the clustering coefficient C, it is basically defined as the likelihood that two 
neighbors of a node are linked themselves, that is, for the i-th node, 



2Ei 



(17) 



Ci = 



Zi{Zi - 1) ' 



where Zi is the number of nearest neighbors of i and E{ is the number of links connecting 
any couple of neighbors; when Ei equals its upper bound Zi(zt— l)/2, the neighborhood 
of i is fully connected. The global clustering coefficient then reads as 



A clustering coefficient close to 1 means that the graph displays a high "cliquishness" , 
while a value close to means that there are few triangles. 

It is easy to see that for the Erdos-Renyi graph, where each link is independently 
drawn with a probability P, the average clustering coefficient is Cer = P '• Therefore, 
for our network, we measure C and we compare it with the average link probability 
Piink; results obtained for different choices of 9 are shown in Figure [3j 

First, we notice that, for a given system size N, the behavior of C and of Cer, with 
respect to 9, is markedly different (see the inset): the latter decreases monotonically 
due to the analogous decrease of the link probability, while the former exhibits two 
extremal points at a relatively large degrees of dilution. In fact, as long as the networks 
are highly connected, the disappearance of a few links yields, in both cases, a modest 
drop in the overall cliquishness. On the other hand, when dilution is significant, the 
intrinsic structure of the "Hebbian graph" matters: as patterns get sparser and sparser, 
surviving links are those connecting nodes whose related patterns display matching with 
non-null entries. In this way, the neighbors of a node are also likely to be connected 
[25| [T71 1261 [27] and the clustering coefficient grows. Finally, at a very large degree 
of dilution, the system approaches the fully-disconnected regime and the clustering 
coefficient decreases. 

In order to compare more effectively our graph and an analogous ER graph, we 
also considered the ratio C/Cer (see the main figure). Interestingly, for 9 relatively 
large, as N gets larger this ratio grows confirming that the few links remaining are very 
effective in maintaining the cliques. This can be understood as follows: to fix ideas let 
us take 9 = 1, so that the average number of non-null entries in a string is L / y/(2N) 



1 N 



(18) 
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Figure 3: Inset: Mean clustering coefficient C (continuous line) for different choices of the parameter 
while N — 1600, 7 = 2 and a = 0.5 are kept fixed; the mean has been performed over all nodes 
making up the graph and over 10 2 realizations. An analogous ER random graph is also considered and 
the related clustering coefficient Cer (dashed line) is shown for comparison. Notice the qualitative 
different behaviors of C and Cer- Main figure: contour plot for the logarithm of the ratio C/Cer, as 
a function of N and 9. Notice that, although for 9 close to 1 both graphs are sparse, C ^> Cer- On 
the other hand, for 9 > 1, both coefficient converge to zero, in the thermodynamic limit; the non-null 
values appearing in the figure are due to finite-size effects. 
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which equals 7a/2 in the high-storage regime under investigation. For simplicity, let 
also assume that ja/2 « 1, and that this holds with vanishing variance for all nodes. 
Therefore, if the node % has k neighbors, its (local) clustering coefficient is either 
(if k < 2) or 1 (if k > 2). Hence, the expected local clustering coefficient can be 
estimated as the probability for a node to display k > 2 nearest neighbors, namely 
1 — (1 — q) 1 — Nq(l — q) N ~ 2 , where <? = 7/ (2iV) is the probability that the pattern of 
an arbitrary node j 7^ i has the non-null entry matching with the one of £j. With some 
algebra we get q ~ 1 — e _7//2 (l + 7/2), which remains finite also in the thermodynamic 
limit, in agreement with results from simulations. For 9 > 1, q — / -/N 1 ~ e /2 and q 
converges to zero. 

4 The statistical mechanics analysis 

In this section we study the thermodynamic properties of the system introduced: At 
first we show its equivalence to a bipartite spin-glass and figure out the order pa- 
rameters of the theory, then we define an interpolating free-energy which generalizes 
the multiple stochastic stability developed in |T3]; this technique allows to obtain the 
replica-symmetric solution in form of a simple sum rule. As a last step, we extremize the 
free energy finding self-consistencies for the order parameters, whose critical behavior 
is also addressed. 

4.1 The equivalent diluted bipartite spin-glass 

As we deal with a structure whose average coordination number may range in [0, N], 
from a statistical mechanics perspective, we aim to define the normalization constant 
D for the Hamiltonian in Equation ([T]), in such a way that its average (which defines 
the extensive energy of the system and is denoted symbolically with the brackets) is 
linearly diverging with the system size, namely (H) oc N. 

By a direct calculation, it is possible to show that this condition is fulfilled by 

D = N 1 - 9 , (19) 
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so that, using the explicit definition for the couplings, we can write 



N L , N 



H 



i<j fi i<j 



For a single realization of the disorder encoded in the memories, the partition func- 
tion reads off as: 

ZnM = E ex P { Jl=e E E tftf^i } • (21) 

Note that, as usual in the Hopfield model, the diagonal term gives an extensive con- 
tribution to the partition function. In the above expression we neglected this diagonal 
term, directly by adding it as a term ^(^^jp) = to the final expression of the free 



energy [12] (see Equation (45)). 

Now, we can introduce another party made up of L soft spins {z^}, namely i.i.d. 
variables with an intrinsic standard Gaussian distribution jV[0, 1], that interact only 
with the original party of binary spins {a{\ via the couplings {£f }; the related partition 
function is 



N L 



Zn,M = E / II MzJ exp U j^r B E E > (22) 

with d^i{z^) standard Gaussian measure for all the z^. By applying Gaussian integra- 
tions as usual [TF], it is easy to see that Zn,l{P] and Zn,l(0] 6) are thermodynamically 



equivalent. The advantage of the expression (22) is that it is linear with respect to the 
memories £f , so that the bare topology is simply that of a bipartite random graph with 
link probability p link = (1 + a)/2, like in [25] . 

Taken O as a generic observable, depending on the spin configurations {a, z}, we 
define the Boltzmann state wg(O) at a given value of (fast) noise /3 as 



uj p {0) = Z N ^f3-i)- 1 Y J J HdtM( Ztl )0({a,z})e 



13 v^L >u - 

f i-e a,! s ff ! z c (23) 



M m 



and we introduce a product space on several replicas of the system as fl s = oj\ (£) u 2 ••• u s 

For a generic function of the memories F(£), the quenched average will be defined by 
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the symbol E and performed in two steps: first we fix the number I of links between the 
two parties and we perform the average over the Gaussian distribution of the memories: 



fino] = / n -j=^ no = / m <wo = m ; (24) 

(i,/i)=l 

then, we perform the average over the binomial distribution for the number of links, 

so that E = E,E[° . Indeed, for example, E[£f] = and E[(£f ) 2 ] = (1 + a)/2. 

Moreover, as we will see, for a natural introduction of the order parameters, it is 
useful to define the number of links, /, as the product of two independent binomial 
variables 

Z = y x , (26) 
where the symbol = stands for the equality in distribution and 

(N\ fl + a lv ll-a N ' lv 

Pi » ~ (l)v 2 V 2 • < 27) 



Of course, a product of two binomial variables is not a binomial variable itself, so at 
finite size this definition is not consistent; nevertheless, in the thermodynamic limit, 
the central limit theorem ensures that only the first two momenta of the distributions 
survive so that the definitions become consistent. 

We also use the symbol (•) to mean (■} = Efi(-) and (-) G = K^Q(-). 

The main thermodynamical quantity of interest is the intensive pressure defined as 

A(a,(3) = lim A N (a,0) = -(3f(a,(3) = Urn -J-ElogZ^ftO . (29) 

Here f(a,(3) = u(a,(3) — /3 _1 s(a,/3) is the free-energy density, u(a, 0) the internal 
energy density and s(a, (3) the entropy density. 

Finally, we define two infinite (in the thermodynamic limit) sets of order parameters, 
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the restricted overlaps, as 



In 1 1 2 

pi x 2 = r E z >l > ( 3 °) 

which define the overlaps (restricted on sub-networks) between two replicas made up 
by parties with l v and l x nodes, respectively. 

4.2 Free energy interpolation and general strategy 

In what follows we assume that no real external fields (as magnetic inputs or partial 
information submission for retrieval) act on the network, but fields insisting on each 
spin are strictly generated by other spins. Thus, the overall field felt by an element of 
a given party is the sum (weighted through the couplings), of the states of the spins 
in the other party. Note that spins are connected in loops using the other party as a 
mirror, therefore, the equivalent analogue neural network is a recurrent network. 

In this section we show that the free-energy can be calculated in specific cases (e.g. 
at the replica symmetrical level) by using a novel technique that has been developed in 
[T3] for fully connected spin-glass models and extended in (25] to diluted ferromagnetic 
models. This technique introduces an external field acting on the system which "im- 
itates" the internal, recurrently-generated input, by reproducing its average statistics. 
While the external, fictitious input does not reproduce the statistics of order two and 
higher, it represents correctly the averages. These external inputs are denoted as rj and 
X (one for each spin in each party) and are distributed following the Gaussian distribu- 
tions with zero mean and whose variances scale according to the underlying topology 
(as a function of a, 9, 7) and coherently approaches zero when the network topology 
disappears. 

In order to recover the second order statistics, the free-energy is interpolated smoothly 
between the case in which all fields are external, and all high order statistics is missing, 
and the case in which all fields are internal, describing the original network: Following 
the original Guerra's schemes [2H1 EES 29, 30J, this allows a powerful sum rule. We use 
an interpolating parameter t G [0, 1] for this morphing, such that for t = the fields 
are all external and the calculation straightforward, while for t = 1 the original model 
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is fully recovered. 

In what follows, for the sake of clearness, we write A = A^(a,P) even though a 
should be introduced only once the thermodynamic limit has been performed. The 
interpolating quenched pressure Ajf(cx,^,t) at finite N is then defined as 

A N (a,P;t) = ^ElogJ] j JJc^(z M )exp ( y/i ]/^Ze J • 

{a} M \ i,ju / 

■ exp ( b y/l-t ^2 a iVi ) exp ( c \/l -t ^ ] exp ( -(1 - t) zjfl,, J . (31) 



Throughout the paper, we assume that the limit A(a, (3) = liniAr_ s . 00 A^(a, (3) = lini7v-!.oo A^(a, (3, t 
1) exists. The "interpolating fields" distributions are chosen to mimic the local fields 
behavior, so that rji, an d 0^ have zero value with probability y(l — a)/2, while, 
with probability y(T-j- a)/2, are normally distributed, except for # M which assumes 
value Consequently, the number of active fields follows Equation (27). As for the 



constants b, c, d, they have to be chosen properly, as shown in the following. 

The strategy for the evaluation of the pressure of the original model, A(a,/3,t = 
1), is to compute the i-streaming of A(a,{3,t), namely d t A(a,fl,t = 1), and use the 
fundamental theorem of calculus to obtain 

A N {a,P) = A N (a,P]t = l) = A N {a,P,t = 0)+[dt'(^A N (a,(3]t)) . (32) 



dt 



t=t' 



When evaluating the streaming d t A, we get the sum of four terms (A, B, C, V); each 
comes as a consequence of the derivation of a corresponding exponential term appearing 

2 Indeed, the presence the field 9^ has much less physical meaning but simplifies the calculations. 
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into Equation (31). In order to proceed we need to compute them explicitly: 

11 11 



A 



N 2VtN( 1 - 9 )/ 2 



— rr ZiWitJi, z u ) = — E — -= — - d^uias, z u ) 
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l, (X 



1 6 lv ' lx 



l, IX 



1 p 



N2NQ-- 

1 g 
N2N( 1 ~°) 



NL 



(^) 



4> - (<?!>! x 2 > 



a/3 7 



(33) 



where in the first passage we used integration by parts and, in the fourth, the fac- 
torization properties of the quenched averages [25], EH |32j |33j EI] (which should be 
understood in the thermodynamic limit). 

The same procedure can be used in the computation of the other terms, so to get: 



B 



\-4k-E- 2 ( 



i b a 

— E ; > SL 

o 2 



N 2 

6 2 /1 + a 
' "2 V 2 



2N 



l-<fl55>)-^f(l- <A >); 



(34) 



C 



— — E 



iv 2 v / r^t 



:E 



1 c 2 ' x 
— E— 
N 2 



2-i E w (*-E w2 Wi 2.v 



ac /1 + a 



ac / 7 
'^"V2 



Z x V fx 

iV-l - (pfc)) ; (35) 



1 w ' x 

N 2 ^ y ^' 



ad /l + 
2 \~Y 
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(36) 



Now, the t-streaming of the pressure reads off as 



N 



[a,(3,t) 



dt 



a(3 



N 



2 



1 + a 



1 + a 
2 

1 + a 
2 

~2 ' 



ac 
~2~ 



1 + a 
2 

b 2 
+ 2 



ad 
~2 



1 + a 



1 + a 



(<&> + 



<- 2 > + 



1 + a 



(Pil) + 



(37) 



4.3 Replica symmetric approximation and fluctuation source 

As it is, this streaming encodes the whole full replica-symmetry-breaking complexity 
[2111 ES] of the underlying glassy phase and it is intractable. Our plan is to split this 
derivative in two terms, one dealing with the averages of the order parameters and one 
accounting for their fluctuations. To this aim we introduce the source of fluctuations, 
S(a, (3, t), as 

n,R f 1 -I- n \ 

(38) 



with 



S(a,M = -^(^)iV e ((g^ 2 -g)(^ 2 -p)), 



q = E» q l \ v = ^iJ v ■ 



(39) 



Notice that the main order parameters q and p sum every overlap, each with its relative 
weight, on every possible subnetwork of the whole network according to the approaches 
[18, 19J and that they recover the standard order parameters of the Hopficld model 
when dilution is neglected [221 [13] . 



In order to relate Equation (38) to Equation (37), let us remember that we still have 



free parameters that can be chosen as0 




N 6 p = 



7\ 4 



m. 



l~q) = p(lY Nl(l-q) 



(40) 



5 In particular, we choose d to cancel the (z 2 ) terms appearing in the first line of equation (37) 
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so to get 

dA(a, t) _ ft + *l n+a \ _ = ft + a^_ ( _ _ ^ 



In the replica symmetric approximation, the order parameters do not fluctuate 
with respect to their quenched average in the thermodynamic limit as they get delta- 
distributed over their replica symmetric averages q, p, which have been denoted with a 
bar. As a consequence, within this approximation, we can neglect the fluctuation source 
term S(a,f3,t) and keep only the replica symmetric overlap averages in the expression 



(41) such that its integration is trivially reduced to a multiplication by one. 

In order to obtain an explicit expression of the sum rule ( J32| , we can then proceed 
to analyze the starting point for the "morphing", namely A(ot,/3;t = 0), which can be 
calculated straightforwardly as it involves only one-body interactions: 



A N (a,P,t = 0)= (42) 

i r ( N L d L \ 

= n E log Yl / II d ^ z ^ exp ( b Yl a ^ + C Y1 z vX# + 2 Y d » z l) = 

= \J\ N ~ 2 lo S 2 + \J\ N ~°* f <Wl°gcosh(V^(!) 4 iV f V) + 

^ 7T^ |Q)'iV-!lo g (l-^l)'ivl(l- ? ) 

4 l-/3(f) 5 ATf(l-g) 2 V2/ V V2/ 

where we used 

1 ( N 

-E^E^ log J2 exp 16 °^ 

{a} \ i 

1 - I 

= -E^E^log2cosh(6 m) = -E lri l v E„ log2cosh(6 77) = 

i 

log2+ I — - — / ajjiyq) log cosh(o rj), (43) 
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and 



j^ijX* 1o s / II d ^ ex p ( c J2 ) ex p ( ^Yl d » 

fj, \ n / \ n 

= -E, E'* | V ^ | + —Ei E 1 *- Vloe ( - 

N \ 2 ^ " d9 ^ J N 2 V 1 _ 





1 + a 



log(l-d) 



1-^(2)^^1(1-5) 



f (|) 3 iHK* I! 



/? (|) 5 JV'(I-S)) (-44) 



Now, substituting the expression for Ajs[(a,/3,t = 0) of Equation(42) into (32), we 



obtain the replica-symmetric free energy (strictly speaking the mathematical pressure) 
of the network as 



i$ 5 (a,/3) = A N (a,P,t = 0) + 



dA% s (a,(3,t 



dt 



t=o 



7 iV"f 



2 

a/37 Q 



log 2 + j djj,(rj) log cosh ( a/ a/3p ^ ^ j 4 iV 4 77 



+ 

i 

7\ 2 



i aPj. , a/37 Ar - 



(45) 



Despite the last expression is meant to hold in the thermodynamic limit, with a little 
mathematical abuse we left the explicit dependence on iV to discuss some features of 



the solution: Equation (45) may look strange due to the strong presence of various 



powers of the volume size N, which in principle are potentially unwanted divergencies. 
We start noticing that, in the limit of zero dilution 6 = and homogeneous distribution 
of fields 7 = 2, the expression for the free-energy recovers the replica symmetric one 
of the analogue Hopfield model [13] (or digital one without retrieval |3j). Moreover, 
remembering the various topological regimes outlined in Section 3, we see that when 
the network changes the topological phase, for instance moving from a fully connected 
topology to a sparse graph, the coordination number may scale with the volume size 
or remain constant. These situations are deeply different from a thermodynamical 
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viewpoint because, in order to have no negligible contributions to the free-energy, fields 
obtained by an extensive number of (finite) terms in the fully connected scenario must 
be (possibly) turned into fields obtained by a finite number of (infinite) terms in the 
dilute regime. As the topology changes, the fields must follow accordingly, which is 
equivalent to a (fast) noise rescaling with the volume size that is another standard 
approach to diluted network [361 E2J EZ] . 

The physical free-energy is then obtained by extremizing this expression with respect 
to the order parameters; we only stress here that, as a general property of these neural 
networks/bipartite spin-glasses, the free-energy now obeys a min-max principle, which 
will not be deepened here (because it does not change the following procedure and it 
has been discussed in [13J). As a consequence, the following system defines the values 



of the overlaps (as functions of a, f3) that must be used into Equation (45) 



dA 
dq 

dA 



am: 



(1 



/3(?)W(1 



dji{rj) tanh 2 (iy/ a (3p 



7 



by which 



q = / dfj,(r)) tanh 2 



V 



(46) 



(47) 



1_ 0(2)3jV3(l -q) 

All the related models (e.g. Viana-Bray |38J, Hopfield [8], Sherrigton-Kirkpatrick 
[39J) display an ergodicity breaking associated with a second order phase transition and 
presence of criticality. If we assume the same behavior even for the model investigated 



here, the self-consistency equation (47) can give hints on the critical line (in the pa- 
rameter space) where ergodicity breaks down. In fact, when leaving the ergodic region 
(implicitly defined by q = 0, p = 0) the order parameters start growing (implicitly 
defining the critical line as the starting point) and, as continuity is assumed through 



the second order kind of transition, we can expand the r. h. s. of Equation (47) for 



low q and obtain a polynomial expression on both sides. Then, due to the principle of 
identity of polynomials, we can equate the two sides term by term obtaining 



A: 



1 



(48) 



which is the critical surface of the system. 
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Mirroring the discussion dealing with the free energy, we note that this result too 



is clearly a consequence of the choice (19) for the normalization factor that gives us 
an extensive thermodynamics. If we normalize choosing D = N, as it is usual in 
the Hopfield model [3], we obtain (turning to T — 1//3 which is most intuitive) T c = 
N e / 2 (l + y/a) -1/7/2 (and recover the AGS line for 9 = and 7 = 2), such that the overall 
effect of increasing dilution is to reduce the value of the critical temperature because the 
couplings, on average, become weaker. In particular, in the finite connectivity regime 
(9 = 1), the network is built of by N links instead of N 2 which, roughly speaking, 
implies a rescaling in the temperature proportional to y/N (coherently with a spin- 
glass behavior), as in the ferromagnetic counterpart its rescale is ruled by N instead of 
yN [36] because the latter is a model defined through the first momentum, while the 
former by the variance. 

Furthermore we stress that the system displays only one critical surface splitting 
the ergodic region from the spin glass and there are no further 'weak-transitions" for 
each sub-overlap, coherently with the scenario discussed in [40] for the similar case of 
the Viana-Bray model [38J. 



5 Fluctuation theory and critical behavior 

The plan of this section is studying the regularity of the rescaled (and centered) overlap 
correlation functions. 

The idea is as follows: If the system undergoes a second order phase transition, the 
(extensive) fluctuations of its order parameters should diverge on the critical surface 



(48), hence they should be described by meromorphic functions; from the poles of these 
functions it is possible to detect the critical surface. As a consequence, an explicit 
knowledge of these functions would confirm (or reject) the critical picture we obtained 
through the small overlap expansion of the previous section. However, obtaining them 
explicitly is not immediate and we sketch in what follows our strategy. At first, we 
define the (rescaled and centered) fluctuations of the order parameters as 

Q l : b = 

Pll = v^fe-pM, (49) 
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such that, while gjj G [-1, +1], p« b G [-1, +1], G E, G E, hence, the square of 
the latter may diverge as expected for second order phase transitions. 

Nevertheless, obtaining them explicitly from the original Hamiltonian is prohibitive 
and we use another procedure, originally outlined in [30]: We evaluate these rescaled 
overlap fluctuations weighted with the non-interacting Hamiltonian in the Maxwell- 
Boltzman distribution, hence (Q^i 2 )t=o, (Qi v i2Pi x iz)t=o, (Pl x i2)t=o, then we derive the 
streaming of a generic observable O (that is in principle a function of the spins of 
the parties and of the quenched memories), namely dt(0)t such that we know how to 
propagate (O)( t=0 ) up to (0)( t= x) (which is our goal), and finally we use this streaming 
equation (which turns out to be a dynamical system) on the Cauchy problem defined by 
(Qf 12)^=0 > (Qi v i2Pi x i2)t=o, {Pf 12) t=Qi obtaining the attended result. Once the procedure 
is completed, the simple analysis of the poles of (Q] vl2 )t=i, (Qi v i2Pi x 12)4=1, (^f x i 2 )t=i 
will identify the critical surfaces of the system. 

Starting with the study of the structure of the derivative, our aim is to compute the 
t-streaming for a generic observable O s of s replicas. Calling 





s 



N i-e 





(50) 



such that 



(0)t 



/n^M^)E CT exp(-^) 



its t-streaming is 



d{O s ) t = d E E W /II M rfM^)0 8 e- 
dt dt E W /n M ^(^)e H 



dH 

~dt 



) 



E Q(O s )n 



dH 

~dt 



) 



(51) 



In the last equation eight terms contribute. Let us call them A\, Bi, Cx, Di, A 2 , B- 



2, 



23 



C2, D 2 and compute them explicitly: 
A 1 ■ 



2 V tN 1 - 6 



E E/ nwEE^ 



2 V tN 1 - 



E E / nw.EE^% 



2N i-e 

g 

2iV 1 -" 



E 



E 



5 ^?7)^X 



£ E -EE 



a,6 i,jx 



a,b %,[X 



fi (°^ a ) 2 ) + E l vWo s q l ^p l : b ) - * y x E Ai^p 



a /i 



a^6 



£<0.(*°) 2 > + E( ^a^) - S E(^4l,aPl+l,a> 
, a a^b a 



{<x} M 



2 V tiV 1 - 
ft 



mo s j2 / n^wEE^M; 



2N i-e 



M m 

5 ^s^X 



E E -EE ^uim^zD 



:E 



a,b i, /i 



£ £ n(o.*T V 2 ) + E "(o.)n(^) 2 ) + 

a<6 /i a 



E 



s r2(O s g s ^ ljS+2 p s + liS+2 ) 
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With analogous calculations 
B 1 - 
Bo -- 



N 



Qs+la ) + s{O s ) 



a<b 



b 2 fl + a\ 5 



N 



s(O s q l s \ w ) + s(O s ) 



(54) 
(55) 



Co 



c 2 /1 + a 
— —G: I — - — 
2 V 2 



N 



S +1X2\ + jj 



S (O s (^ +1 ) 2 ) - 



{O s p% 1:S+2 } 



(56) 
(57) 



Do 



daN /1 + a 



2 V 2 
daiV /1 + a 



a 

S (O s (^ +1 ) 2 ) . 



Therefore, merging all these terms together, the streaming is 



d_ 
dl 



(0.)t 



is s 



+ 



\a<b 

s(s + l) 



(OQ ' 8+1,8+2^8+1,3+2) t 



(58) 
(59) 



(60) 



In order to control the overlap fluctuations, namely (Q 2 12 )t=i, (Qi v i2Pi x i2)t=i, (Pi x i2)t=i-> 
noting that the streaming equation pastes two replicas to the ones already involved 
(s = 2 so far), we need to study nine correlation functions. It is then useful to introduce 
them and refer to them by capital letters so to simplify their visualization: 



(Ql v 12)t 
{Ql v 12Pl x 12)t 
(Pui2)t 



A(t), (Q M2 Q M3 >t = B(t), (Qi vl 2Qi v 34)t = C(t), (61) 

D(t), (Q lv i 2 Pi x n)t = E(t), (Qi v i 2 Pi x u)t = F(t), (62) 

G(t), (P lxl2 Pi x i3)t = Hit), (P, xl2 P, x3 4) t = lit)- (63) 
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Let us now sketch their streaming. First, we introduce the operator "dot" as 

6 2 d0 



/3y / «7 dt 



which simplifies calculations and shifts the propagation of the streaming from t — 1 
to t = 0^/a r y/2. Using this we sketch how to write the streaming of the first two 
correlations (as it works in the same way for any other): 

A = (Q? l) 12 ( 5^12-P/ X 12)t - 4(Qf ?jl2 (5z lJ 13-Pz x 13)t + 3(Q^ 12 Qi^ 3 4-P/ x 34)t, 
B = (Ql v 12Ql v 13 (Ql v 12Pl x 12 + <5/,,13-P« x 13 + <5/ t; 23-P« x 23^ )t ~ 

— 3(<5i 7; 12-Pi x 13 (Ql v uPl x 14 + Ql n 24,Pl x M + Q l^Pl^i)) t + 6(Ql v 12Pl x 13Ql n 4sPl x 4$&) 

By assuming a Gaussian behavior, as in the strategy outlined in [30], we can write the 
overall streaming of the correlation functions in the form of the following differential 



system 






A = 


IAD - WE + QCF, 




B = 


2AE + 2BD - ABE - 6BF - 6EC + 12CF, 




c = 


2AF + 2CD + 8BE - 1QBF - 1QCE + 20CF, 




D = 


AG - ABH + 3CI + D 2 - AE 2 + 3F 2 , 




E = 


AH + BG- 2BH - 3BI - 3CH + 6CI + 2ED - 2E 2 


- 6EF + 6F 2 , 


F = 


AI + CG + ABH - 8BI - 8CH + 10CJ + 2DF + AE 2 


- IGEF + 10F 2 , 


G = 


2GD - 8HE + GIF, 




H = 


2GE + 2HD - AHE - 6HF - 61 E + 121 F, 




i = 


2GF + 2DI + 8HE - 1GHF - 1GIE + 20/F. 


( 



As we are interested in discussing criticality and not the whole glassy phase, it is possible 
to solve this system starting from the high noise region, once the initial conditions at 
t = are known. As at t = everything is factorized, the only needed check is by the 
correlations inside each party. Starting with the first party, we have to study A,B,C 
at t = 0. As only the diagonal terms give non-negligible contribution, it is immediate 
to work out this first set of starting points as 
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A(0) = (Ql 12 ) = N(((&) 2 ) ~ 2E^(^ 2 > + ^ = 

5(0) = (Q M2 Q M 3> = N((q[lq[l) - f) = yJ^Nlq - Nq 2 , (67) 

c(o) = (g M2 Q M4 ) = - 1) = * (Ox>W>M> - Nt = 

^'H( i-!g?-J - tf (68) 

For the second party we need to evaluate G, H, I at t = 0. The only difference with 
the first party is that z 2 ^ 1 as for the cr's. 

G(0) = (/£ 12 > = N(((p[l) 2 ) - 2E lx p l *{p 1 *) + E lx p 2 x = 

= ^K^E((^) 2 )c((^) 2 )c-P 2 ) = (a^N^^-Nf, (69) 



H(0) = (P lx nPi x is} = N((p l * 2 p%}-f) = (a|r f iV 9/2 u(z) u(z 2 ) - Np 2 , (70) 

1(0) = (^12^34) = tf«pfeffe> - P 2 ) = (^P^ /2 " 2 (*) - ^ 2 - (71) 

Now, oj(z 2 ) and w(z) are Gaussian integrals and can be explicitly calculated as 

,, M frf//(*)*exp(6*x+|* 2 ) _ b( X ) _ , . 

U{Z) ~ fdv(z)e W {bz X + lz*) "l-d~ U ' 

JrfM^ 2 exp(fe X + ^ 2 ) _ l-rf + fe 2 ( X 2 ) 
1 J / rf/i^) exp {bz X + lz 2 ) {l-df " 

I- pJlN e l 2 {l-q + ap) 



(l-py/?N°/*(l-q)y 



Finally, we have obviously D(0) = E(0) = F(0) = 0, because at t — the two parties 
are independent. As we are interested in finding where ergodicity becomes broken (the 
critical line), we start propagating t (from to 1) from the annealed region (high noise 
limit), where q = and p = 0. It is immediate to check that, for the only terms that 
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we need to consider, A, D, G (the other being strictly zero on the whole t 6 [0, 1]), the 
starting points are: 

A(0) = = -, (74) 

V 7 r 

D(0) = 0, (75) 

N-2 1 

GO = — = — . (76) 

^(1 - /3,/fiVt) 2 rs 2 



Where we have defined r = \jl,N 2 ; $ = 1 — /3^ t y^N "2 . 
The evolution is ruled by 

A = 2AD (77) 
D = AG + D 2 (78) 
G = 2GD. (79) 

Noticing that g = by substitution, and that ^© = s 2 we obtain immediately : 

A(f ) = G(t) S 2 = G(t) (l - /?yfiV^ 2 . (80) 

The system then reduces to two differential equations; calling Y = D + Gs, we have 
y = D + Gs = GV + D 2 + 2GDs = Y 2 with solution Y(t) = p^L, and y(0) = 
D(0) + G(0)s = ^ by which we get 

Y(t = yfcfP-) = 1 - 7 1 ^, (81) 

Vapors) v ^ A r- e/2 / 1 _ / 5 v ^ ive /2( 1 + v ^ ) 

i.e. there is a regular behavior up to 

& = ^ fl 1 , (82) 

Tivf(i+^) 



which confirms the result obtained in Equation (48). Now, we can consider separately 
the evolution equation for G and D: 

G = 2G(t) (Y(t) - sG(tyj = -^—^G(t) - 2sG(t), (83) 



where we used Y(t) = (rs — t) 1 . Dividing both sides by G 2 and calling Z = G 1 we 
get an ordinary first order differential equation for Z(t): 

- Z(t) = 2Y(t)Z(t) -2s = —^—Z(t) - 2s. (84) 

rs —t 

that have the following solution for the initial condition Z(0) = rs 2 : 

Z(t) = 2s(rs-t) --(rs-t) 2 . (85) 



r 



From Z(t) we obtain G(t), that is, 



G ® = ~( *W -Lty (86) 

{rs-t){s + -) 



Using Equation (80) and remembering that D{t) = Y(t) — G(t)s, we obtain the other 



overlap fluctuations 



{Qi n i2) = t \-/ \-> (87) 



(Qi v i2Pi x i2) - t yt \~' 

1 - (3JIN°/ 2 {1 + y/a) ) ( 1 - PyftNW(\ - y/a) 



[Pin) = 7 ^-77 n-- (89) 

IN' 6 / 2 I 1 - ^N e / 2 (1 + y/a) J ( 1 - PJ\N°I\1 - y/a) 



A simple visual inspection of the formula above allows to confirm that the poles are 
located at 

Ff 



13 V2 m (1 + ^ ) = 1 ' 
confirming the heuristic result previously obtained. We can easily see furthermore that 
in the fully connected limit (7 = 2 and 6 = 0) we recover the result of [3]. 



6 Conclusions and outlooks 



In this paper we introduced and solved, at the replica symmetric level, two disordered 
mean-field systems: the former provides a generalization of the analogue neural network 



by introducing dilution into its patterns encoding the memories, the latter is a bipartite 
and diluted spin-glass made up of a Gaussian party and an Ising party, respectively. 
From an applicative viewpoint (not discussed here, see e.g. [4 1 J ) , the interest in these 
models raises in different contexts, but their peculiarity resides in the existence of 
sparse entries (instead of classical dilution on the neural network links as performed 
for instance earlier by Sompolinsky [12] through random graphs or recently by Coolen 
and coworkers [18] through small-worlds or scale-free architectures) which allows, when 
possible, parallel retrieval as for instance discussed in |41j . Interestingly, as we show, 
the Hamiltonians describing these systems are thermodynamically equivalent. 

In our investigations we first considered the diluted analogue neural network and 
focused on the topological properties of the emergent weighted graph. We found an 
exact expression for the coupling distribution, showing that in the thermodynamic limit 
it converges to a central Gaussian distribution with variance scaling linearly with the 
system size N. We also calculated the average link probability which, as expected, 
depends crucially on the degree of dilution introduced. More precisely, by properly 
tuning it, the emergent structure displays an average coordination number z which can 
range from z = N (fully-connected regime) to z = O(N) (constant link probability), 
to finite with z > 1 (overpercolated network) or z < 1 (underpercolated network). 

Then, we moved to the thermodynamical analysis, where, through an interpolation 
scheme recently developed for fully connected Hebbian kernels [13] , we obtained explic- 
itly the replica symmetric free-energy coupled with its self-consistency equations. The 
overlaps, order parameters of the theory, turn out to be classical weighted sums of sub- 
overlaps defined on all possible sub-graphs (as for instance discussed in [25l[T9] ). Both a 
small overlap expansion of these self-consistencies, as well as a whole fluctuation theory 
developed for their rescaled correlations, confirm a critical behavior on a surface (in 
the a, (3, 7, 8 iperplane) that reduces to the well-known of Amit-Gutfreud-Sompolinsky 
when the dilution is sent to zero [3] . On the other hand, the net effect of entry dilution 
in bitstrings (which weakens the coupling strength) is to rescale accordingly the critical 
noise level at which ergodicity breaks down, as expected. 

Without imposing retrieval through Lagrange multipliers (as for analogue patterns it 
is not a spontaneous phenomenon, see [13]) the system displays only two phases, an 
ergodic one (where all overlaps are zero) and a spin-glass one (where overlaps are 
non-zero), split by the second order critical surfaces (over which overlaps start being 
non-zero) which defines criticality. 

Outlooks should follow in two separa]^ directions: from a pure speculative and 



modeling viewpoint attention should be paid to the mechanism of replica-symmetry 
breaking, which is known to happen on these models, while for an applicative perspec- 
tive they could be integrated in the theoretical framework where collocate the plethora 
of experimental results stemmed from complex system analysis, e.g. from neural and 
immunological contexts. We plan to report soon on both the topics. 
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